Smart survey with progressive discovery

ABSTRACT

Embodiments of the invention collect data or information from a dynamic and adaptive target group selection. Sets of experts are selected that each have higher value metrics scores relative to scores of other, unselected experts of a population of known experts, and wherein the metric values are indicative of an expertise relevant to one or more questions in a survey. Responses to the survey are used to expand the set of experts by adding experts noted in answer referrals, and to automatically update the expert value metric scores as a function of response timeliness and of degrees of correlation of answers to an expected answer or other norm. The survey is updated by removing satisfied questions, and expert selections, answers analysis and survey updating and resending steps are iteratively repeated until each question is satisfied, wherein the expert sets are dynamically revised for each iteration.

BACKGROUND

The present invention relates to systems and methods for collecting dataor information from a population of people.

Currently, the common approach to complex data discovery is to send alarge survey with potentially hundreds of questions to a largepopulation of people who might have some insight that will be helpful.Response rates are often underwhelming, which may lead to low confidencein the data received for several reasons; first, because of the lowpercentage of responses, and also because there may be some unknown skewin those that did respond. For example, if 20% responded, there may besome reason why those 20% felt compelled to respond, such as they arehighly dissatisfied, wherein their responses may not be indicative ofthe responses of the 80% that did not respond, which may be satisfied ornot strongly positive or negative. Also, there is evidence thatnon-response may be driven by an intimidation factor of being presentedwith a large survey with a large number of questions, many of which areirrelevant to a particular person.

BRIEF SUMMARY

In one embodiment, a method is provided for collection of data orinformation from a dynamic and adaptive target group selection. Themethod includes selecting a set of experts that each have higher valuemetrics scores relative to the scores of other unselected experts of apopulation of known experts, and wherein the metric values areindicative of an expertise relevant to one or more questions in asurvey. Responses to the survey are used to expand the set of experts byadding experts noted in answer referrals, and to automatically updatethe expert value metric scores as a function of response timeliness anda degree of correlation of an answer to an expected answer or othernorm. The survey is updated by removing satisfied questions, and updatedsurveys are iteratively sent to updated sets of experts selected fromthe expanded expert sets as relevant to the remaining questions andhaving higher updated metric values, until each question in the updatedsurvey is satisfied and removed.

In another embodiment, a computer system includes a processing unit,computer readable memory and a computer readable storage system. Programinstructions on the computer readable storage system cause theprocessing unit to select experts that each have higher value metricsscores relative to other unselected experts of a population of knownexperts and that are indicative of an expertise relevant to one or morequestions in a survey, and also to expand the set of experts by addingexperts noted in referrals in received answers. Instructions also causethe processing unit to determine a degree of correlation of an answerwithin a response to an expected answer or other norm, and to update theexpert value metric scores as a function of response timeliness and thedetermined degree of correlation to the expected answer or norm.Instructions are also further to update the survey by removing questionssatisfied by response answers and send out updated surveys to selectedexperts until each question in the survey questions is satisfied andremoved, as determined by iterative repetitive application of the other,above instructions.

In another embodiment, a computer program product includes programinstructions to select experts that each have higher value metricsscores relative to other unselected experts of a population of knownexperts and that are indicative of an expertise relevant to one or morequestions in a survey, and also to expand the set of experts by addingexperts noted in referrals in received answers. Instructions are also todetermine a degree of correlation of an answer within a response to anexpected answer or other norm, and to update the expert value metricscores as a function of response timeliness and the determined degree ofcorrelation to the expected answer or norm. Instructions are alsofurther to update the survey by removing questions satisfied by responseanswers and send out updated surveys to selected experts until eachquestion in the survey questions is satisfied and removed as determinedby iterative repetitive application of the other, above instructions.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

These and other features of this invention will be more readilyunderstood from the following detailed description of the variousaspects of the invention taken in conjunction with the accompanyingdrawings in which:

FIG. 1 is a diagrammatic illustration of an embodiment of a system,method or process according to the present invention for collection ofdata or information from a dynamic and adaptive target group selection.

FIG. 2 is a diagrammatic illustration of an embodiment of a system,method or process according to the present invention for evaluatinganswers for satisfaction of questions and for adjusting expert metricvalues and outlier status.

FIG. 3 is a diagrammatic illustration of a computerized implementationof an embodiment of the present invention.

The drawings are not necessarily to scale. The drawings are merelyschematic representations, not intended to portray specific parametersof the invention. The drawings are intended to depict only typicalembodiments of the invention, and therefore should not be considered aslimiting the scope of the invention. In the drawings, like numberingrepresents like elements.

DETAILED DESCRIPTION

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, in abaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including, but not limited to, wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

Choosing recipients for questionnaires may be problematic in prior artmethodology, for example it may take many weeks to determine a contactlist, and this list is often only partially complete and accurate.Further, knowing which of many possible questions are appropriate to askof which contacts requires manpower and resources. Often a defaultapproach is to send an entire questionnaire to all known appropriateparties and hope for the best. Once sent, a common problem with a largequestionnaire is lack of response, since recipients may decide that theydo not have the free time required to provide a large number (evenhundreds) of answers, many of which are not relevant to their positionor skill set. Discovery teams must manually track returns to determinecompliance and data quality, and send reminders to those who aredelinquent and rework requests to those returning incomplete orinaccurate questionnaires. Finally, the discovery team must manuallycorrelate answers in meaningful ways to explore trends and patterns toprovide insight of value about the complex questionnaire environment.

Referring now to FIG. 1, an embodiment of a method or system accordingto the present invention is illustrated for collection of data orinformation from a dynamic and adaptive target group selection. At 102an initial subset of primary expert contacts is automatically identifiedand selected from a larger inclusive organizational or knowledge-basepopulation in response to having higher expert value metrics scoresrelative to value metric scores of other unselected experts of apopulation of known experts, and that are indicative of an expertiserelevant to at least one question in a plurality of survey questions.Thus, a first subset of possible experts may be selected from historicdata indicating previous expert associations relevant to currentquestions, and also evidencing relatively higher metrics that mayindicate higher probabilities of receiving timely, quality responsesfrom the survey questions.

At 104 the plurality of survey questions are sent to the selected set ofexperts. At 106 the system automatically expands the selected set ofexperts by adding experts from the population of known experts that arenoted in referrals in answers received in response to the sentquestions. At 108 the value metric scores of the expanded set expertsare also automatically updated (for example, via a programmable device)as a function of characteristics of the expert's responses to the surveyquestions, for example a timeliness of the responses received fromassociated experts, and of qualities of the responses relative to otherresponses or expected answers (degrees of correlations of answer withinsaid responses to an expected answer, etc.). More particularly,embodiments may update expert value metrics for associated experts inresponse to determining a degree of correlation of their answers to anexpected answer or answer characteristic, for example to a hypothesisanswer or a majority answer, or with respect to a variance from observedclusters of answers, etc.

At 110 the plurality of survey questions are also updated by removingquestions satisfied by response answers from the selected set ofexperts. For example, if a sufficient representation of the populationhas completed a question and/or generated a sufficient number ofresponses having a sufficient correlation/variance with respect to otheranswers, etc.

At 112 the system and process steps thus described iteratively repeatuntil each of the survey questions is satisfied, wherein the processends at 114. More particularly, each iteration at 102-104-106-108-110refines the set of experts receiving further refined sets of surveyquestions. Each new survey iteration comprises questions in the surveythat have not yet been satisfied through previous iterations andresponses, wherein only those unsatisfied questions need be sent out,and only to the new experts selected as a function of their relativemetric values. Additionally, follow-up questions may also be added tothe survey set, for example in response to earlier question answers.Lower scoring experts may also be dropped from the set in favor ofhigher scoring experts, in expectation that the higher scoring expertsare more likely to provide satisfactory answers, and/or to respond, etc.Thus, additional targets, questions and surveys may be chosen,generated, etc. until all questions are satisfied.

Embodiments of the present invention may increase efficiencies byreducing the total number of experts contacted through each surveyiteration targeting questions and surveys to higher-value experts,wherein the corresponding responses may be as good or better in qualityand/or totality compared to prior art techniques that send out surveysto more but lower-quality respondents, which may have higher overallresponse rates but comprising lower overall quality that wastesresources needed to filter out the low-quality, irrelevant results.

New additional experts suggested or noted by returned surveys are taggedwith social rating and quality metrics, or their current metrics areupdated in a scoring process that identifies expert individuals fromcollected survey data received from the initial primary contact subsetin response to the initial baseline survey. The initial baseline surveyquestions may generally prompt the targets to uncover contactinformation for those other individuals having specific expertiserelevant to the baseline survey. As key individuals are identified andcatalogued, they may be assigned a value metric that may be associatedwith the different subject areas, systems or applications that they maybe responsible for, or with which they may be regarded as an expert. Theembodiment thus self-generates a dynamic network of contacts for usewith each survey iteration, wherein a selected set of contacts mayexpand or contract with each iteration, to which targeted questions aresent from a databank of tagged questions. An expert's metric value mayincrease if multiple respondents refer to the same expert as anappropriate person to answer a given question or set of questions, ordecrease relative to another expert for relatively fewer references to agiven item or category.

FIG. 2 is a block diagram illustration of expert metric adjustments andquestion satisfactions determined from survey responses in oneembodiment according to the present invention. At 120 survey responseanswers are compared to expected or mean answers or some other normativestandard, and at 122 accordingly identified as outliers or inliers. Ametric value of the answering expert is accordingly incremented orotherwise adjusted as a result of the outlier or inlier determination at126 or 124, respectively, and wherein the expert may also be labeled asan outlier at 126. At 128 the question associated with the answer isevaluated for retirement, and retired if a question satisfactionstandard is met, for example if a sufficient number of inlier orcombination of inlier and outlier answers has been received, and othercriteria may also be applied as discussed elsewhere in the presentapplication. The metric value of the answering expert is also adjustedas a function of the answer inlier or outlier status at 130.

Metric scores may further include quality criteria. For example, anexpert set may be dynamically composed based on social referral rankingsand also on other criteria. In one embodiment, the timeliness ofresponses is used to raise or lower a value metric, to generally reflecta policy that the quicker they get back with a response, the morevaluable they are as participants and thus more likely to be included insubsequent iterations. Thus, in one example, the value metric score ofan expert is increased or decreased as a function of comparing anelapsed time of a response (from sending the questions to receiving theresponse) to a standard or normal sent response time. Norms may bespecified, or they may be dynamic, for example determining means orother average response times from actual survey responses, andincreasing expert values for quicker responses and lowering values forslower responses, or for failures to respond at all. A different(longer) threshold may be used to determine an allowable time forresponse, wherein exceeding it may result in a failure to respond valueadjustment. Reminders may also be sent to an expert in response to afailure to receive an answer within a threshold time for response. Thus,an expert may be removed from a set of experts in response to a failureto receive a response within an allowable time for response to thereminder, or if a specified number of reminders fails to trigger aresponse, or their metric drops below a threshold value through downwardincrements, and the expert may be “deactivated” for this and/or othersubsequent survey iterations, perhaps flagging the expert anon-responder for a final results report.

Expert metric values, as well as question satisfaction determinations,may also be adjusted or determined as a function of correlation ofanswers to those of other experts in their domain, for example, scoringthose that do not tend to have outlier answers higher than other expertswhose answers trend outside of expected answer norms. Correlations to ahypothesis answer for a question may also be utilized, for examplescoring experts whose answers are within reason to what is expected(within a normative cluster of answers, or within a threshold varianceof expected answers) higher than those other experts tending towardoutlier answers (outside of a cluster or variance).

By tagging each of the population of experts to whom the surveys aresent with social ratings, quality scores and other metrics, the systemmay dynamically revise and maintain expert lists in real-time for agiven question or survey. Thus, in some embodiments, only the top threemetric value experts indicated as relevant to a given question receivethe survey question, wherein this selection may change with each surveyiteration based on updating the expert metrics dynamically anditeratively with each round of question responses.

The invention performs data collection in a smart or intelligent fashionthat learns from early returns and modifies the approach based on avalidation of the information, a validation of the respondents, and arefined targeting of questions to the most appropriate recipientdemographic, and wherein target groups may expand or contract based onreferrals from current recipients. The expert-system survey tool therebyself-generates an expanding population of target subject matter expertsfrom which it solicit responses, the subject selected by referrals fromexperts already in the population of known users. Once enough expertsare found, of a sufficient quality metric rating, the system may alsostop asking for more referrals. Thus, additional experts may beidentified and added to a previous expert set; questions may bedifferently targeted to experts within a current or previous set, forexample, a domain of expertise may be revised for an expert, resultingin different questions for that expert); or experts may be dropped fromthe set, for example through lack of or low rates of cross-referencesfrom others of the set or low quality metrics.

Each survey iteration may reach out to newly included experts, and alsoensure that experts already contacted or responding do not receiveduplicate questions or surveys (for example, decrementing their valuemetric or otherwise dropping them from subsequent survey iterations).Depending on the value of accepted answers, additional survey iterationsmay include generating more detailed questions with regard to an answer,and which could be sent to larger or smaller sets relative to those whoresponded to the higher-level first question: for example, to a subsetof targets who responded most accurately with regard tocorrelation/variance and/or in a relatively more timely fashion, tothose who had the highest cross-reference referral rate from otherexperts, or to those specifically designated as the best to answer acertain question.

Follow-up questions at any iteration may be automatically generated inresponse to previous responses through correlation/variance data. Forexample, “Why is the answer to [question A] equal to [answer 1]”? may beautomatically generated from a review of data listing [answer 1] as theanswer value for the response to [question A], or “Most respondentsindicated that the answer to [question A] is equal to [answer 2], whydid you answer with [answer 1]?”Follow-up questions may also be manuallyadded to a question bank in response to a survey administrator review ofanswers, including a correlation/variances from majority or acceptedanswers. Answers may also be weighted based on the assessed competenceof the person providing the answer, for example their skill orexperience level if related to the question being asked, and/or as afunction or a peer referral ranking.

Questions may be retired from a survey at 114 once a threshold isreached based on a number of answers, a weighting of the one or moreanswers, or when a consensus is indicated by the answers (for example,few occurrences of contradicting answers within an answer set, less thana threshold value).

Thus, embodiments of the present invention provide an expert-systemsurvey tool that adaptively self-generates a dynamically expanding,contracting or otherwise refined population of target subject matterexperts from which it solicit responses. These come from referrals fromexperts already in the population of known users. The numbers of expertsmay be limited to provide further efficiencies; once enough experts arefound, of a sufficient quality rating as indicated by their metricsrelevant to the question, the system may stop asking for more referrals,through application of user or system defined thresholds or otherlimiting criteria. The present invention makes use of expert systemtechniques to progressively discover insights and automatically buildkey individuals that are listed as targeted subject matter experts.These key individuals may be identified based on a tree-based referralsystem that will intelligently direct relevant questions to aself-expanding list of contacts.

The process learns from each return and modifies a survey approach basedon a validation of the information, a validation of the respondents, anda refined targeting of questions to a more appropriate recipientdemographic (which itself expands based on referrals from currentrecipients). Rather than send a large collection of questions to allpotential recipients all at once, processes according to the presentinvention ask a smaller number of key questions, and get responses toprogressively refine or broaden discovery until desired results areachieved in terms of information gathered and confidence level attained.The metrics may also be used to eliminate duplication of requests tocertain individuals or to multiple experts with regard to the samesubject, for example, sending a query to no more than three subjectmatter area experts rather than to every one, and optionally furtherselecting the most historically responsive three of possible qualifyingexperts, further streamlining the automated process of data collectionacross a business or other enterprise comprising multitudes of possiblesurvey targets.

In one aspect, the process may automatically accept referrals fromrespondents to amend a targeted population of recipients based onskill/knowledge qualities, for example Bob Jones replies that “Joe Smithis our resident subject matter expert on topic XYZ, so ask him”,directly and automatically resulting in a subsequent survey query to JoeSmith. Due to the complexity of managing and collecting such data,automated embodiments of the present invention provide a smart surveysystem that will substantially reduce the amount of time and improve theaccuracy of the data collected relative to manual survey methods, whichare generally time-intensive and normally performed by differentindividuals meeting in person, scheduling conference calls, manuallypiecing together data and performing individual or cooperative manualanalysis, etc.

Validation key range and/or type may be applied in real-time to questionanswers, in order to avoid data errors or poor results from inconsistentanswers, etc. Embodiments also track returns and manage reminder andrework requests, for example only resending questions that need betteror more responses. Data inaccuracy may also be reduced throughautomatically filtering or otherwise analyzing answers at input toprevent input of human errors, elements of bias or inappropriate repliesfrom data, rather than later, subsequently identifying and removing sucherrors. Some embodiments filter the data to remove sensitive words orterms objectionable to certain individuals or organizations, or that maybe subject to misinterpretation, before they are entered into databaseor other data storage means.

In another aspect intelligent computer and system data collectionprovides for improved accuracy and lowered risk for performing sensitivetasks which may include collecting confidential information which mayimpose duties upon the collector. By using an automated smart survey tocollect and analyze data, confidential information may be easilyencrypted at input or otherwise protected from subsequent, unauthorizeddisclosure.

In some embodiments, the different target individuals are each providedwith a Uniform Resource Locator (URL) that links to a smart surveycontaining key Extensible Mark-up Language (XML) tags used to identifythem and their value metric (for example, a numeric “expert score”reflecting their relative expert status, timely responsiveness, etc.),and also to store the data within a relational database. Every time asubject matter expert or other key individual is identified in an answeras part of a data collection process, the value metric for the keyindividual may be revised (generally increasing with every positivereference to their name). The value metric may also be associated withspecific parts of a particular enterprise associated with the populationof experts, for example components within an organization's computinginfrastructure, application or computer system, a business process in aservice-oriented architecture, etc., and that has been identified aspart of or relevant to the smart survey.

Notice and other reporting on completion rates may be provided in adashboard or other user interface, and wherein return data may beorganized into a set of visualization graphics to provide a user,discovery team, etc., with a first-pass view of a complex environment.For example, as data is collected, results may be displayed on a networkcomputer dashboard that will allow a user or data collection team toverify the accuracy of the data collected, and also to communicate withthe key individuals displayed that are associated with the differentparts of the infrastructure. In some embodiments, a dynamic web page isutilized to collect the value metrics data and store it in a relationaldatabase, one that enables identifying the key individuals in differentparts of the computing environment. A web service or web component maycommunicate with the smart survey system and update the relevant XMLtags and value metrics for collecting answers and other data during asurvey process.

Some embodiments determine when a result for a survey question has met a“condition of satisfaction” (CoS) so that it may be retired. Questionsthat naturally have a limited number of answers (for example,true/false, or high/med/low, etc.) define a range or type of values forapplication of answer validation processes. For each survey question, aset of responders are identified along with their contact information,generally a statistically significant number of responders for eachquestion, though this is not a requirement. Thus, systems mayautomatically send each responder a URL link, which contains a web pagewith a custom interface that only presents the questions they aretargeted to answer. Each question has both an answer field and a fieldin which the responder can enter additional contacts that they feelmight be better suited to answer that particular question; in someembodiments, numbers of references may be limited, for example up totwo.

As answers are returned for a given question, they are analyzed againstone or more “condition of satisfaction” (CoS) metrics or thresholds.Examples of CoS satisfaction metrics include a minimum threshold numberof responders replying with the satisfying answer, or that a set ofresponses (minus outliers) includes an answer value within a variancevalue or other tolerance, in one aspect validating accuracy of an answerthrough comparisons when the same question generates responses frommultiple contacts, for example through variance and correlationanalysis. Responders whose answers are outliers (for example, fallingoutside a defined general cluster) may be sent a clarification requestto explain why their answer might be significantly different from thenorm. They may also be given an opportunity to re-submit an answer thatis within the norm, which will remove them from the outlier group.

In some embodiments, when a survey question appears to have met itscondition of satisfaction metric, the question is sent with its answersand related statistics to a survey administrator entity for acceptance.The data may also include those “clarification responses” from theresponders whose answers were outside the norm. If the surveyadministrator is satisfied and accepts the results (for example, theanswer meets variance or correlation standards, or an outlier divergenceis adequately accounted for by a clarification response), the surveyquestion is retired from further survey use and the answer is accepted.A survey dashboard may be updated to reflect the completion status ofthe survey question and the partial results posted. In another aspect,respondents to the retired question may have their expert ranking metricupdated, based on factors such as their timeliness, the quality of theiranswer (amount of variance or correlation to a majority or generallyaccepted answer), their referral rate (which takes into account thequality of the referee), etc.

If the process or survey administrator adds a new related question tothe survey, based on the results of the retired question, the newrelated question will enter the system and be processed as a newquestion and posted to the associated target respondents of the retiredquestion. It may also be linked to the retired question such that theretired question and the accepted answer may be displayed (for example,provided in the dynamic web page screen) with the new question, in oneaspect to provide context for the new question.

Further, once a survey is complete, a dashboard may reflect the finalstate of each of the questions and answers, a confidence level in eachof the answers, and various other reports such as the number ofrespondents, the quality value for each respondent, the list of nonrespondents, etc.

By providing for the self-retirement of a question, the presentinvention eliminates the need for manual intervention to edit on-goingsurveys by reducing the survey's bank of questions. Once a question hasbeen responded to in a satisfactory fashion, that question is retired,marked as complete. A variety of standards may be used, and in oneexample the condition for satisfaction is that three domain experts haveresponded with answers that have a moderately high correlation factor,indicating a high confidence in the answer sufficient to eliminate theneed for further expert consideration of the question.

Embodiments of the present invention may also automatically weed outexperts with low ratings. Follow-up queries and confirmation requests toan expert whose answer is an outlier compared to answers from otherexperts may also notify them of their outlier status, which may causethem to rethink their answer in view of its outlier status and allowthem to revise, confirm or defend their opposing view. In one aspect,this feedback may influence the social ranking aspect of an expertmetric, one based on how their answers in general correlate to ahypothesis or to other expert answers. Further, a subset of experts maybe selected to intentionally include social outliers, in order to seekbroader viewpoints or range of answers, or to test strength ofresponses: for example, if outlier answers correlate with those ofnon-outliers, then confidence in the answer may be enhanced, asdiffering viewpoints agree on this answer.

Outliers may also be quickly identified with respect to divergence froma hypothesis answer associated from an initial question, even wherehistoric data does not indicate or provide enough data to determineoutlier status. For example, if the question is, “Please rate theoverall maturity of your change control process between 0 and 5, where 5is a world-class environment”, and the hypothesis answer might be in therange of 1.5-3.0, then initial answers outside of this range may resultin an outlier status, or an outlier metric increment.

Referring now to FIG. 3, an exemplary computerized implementation of anembodiment of the present invention includes client computer or otherprogrammable device 322 in communication with a user interface 328 andwith one or more third party servers 336 accessible through an SSL orother secure web interface 340, for example in response to computerreadable code in a file residing in a memory 316 or a storage system 332within a computer network infrastructure 326. The code, when executed bythe central processor 338, provides expert selector 342, answerevaluator 344, expert metric value adjustor 346 and questionsatisfier/selector 348 components that perform one or more of theprocess and system functions described above with respect to FIGS. 1 and2.

The implementation 326 is intended to demonstrate, among other things,that the present invention could be implemented within a networkenvironment (e.g., the Internet, a wide area network (WAN), a local areanetwork (LAN) or a virtual private network (VPN), etc.) Communicationcan occur via any combination of various types of communications links:for example, communication links can comprise addressable connectionsthat may utilize any combination of wired and/or wireless transmissionmethods.

Where communications occur via the Internet, connectivity could beprovided by conventional TCP/IP sockets-based protocol, and an Internetservice provider could be used to establish connectivity to theInternet. Still yet, the network infrastructure 326 is intended todemonstrate that an application of an embodiment of the invention can bedeployed, managed, serviced, etc. by a service provider who offers toimplement, deploy, and/or perform the functions of the present inventionfor others.

The computer 322 comprises various components, some of which areillustrated within the computer 322. More particularly, as shown, thecomputer 322 includes a processing unit (CPU) 338 in communication withthe memory 316 and with one or more external I/O devices/resources 324,user interfaces 328 and storage systems 332. In general, the processingunit 338 may execute computer program code, such as the code toimplement one or more of the process steps illustrated in the Figures,which may be stored in the memory 316 and/or external storage system 332or user interface device 328.

The network infrastructure 326 is only illustrative of various types ofcomputer infrastructures for implementing the invention. For example, inone embodiment, computer infrastructure 326 comprises two or morecomputing devices (e.g., a server cluster) that communicate over anetwork. Moreover, the computer 322 is only representative of variouspossible computer systems that can include numerous combinations ofhardware. To this extent, in other embodiments, the computer 322 cancomprise any specific purpose computing article of manufacturecomprising hardware and/or computer program code for performing specificfunctions, any computing article of manufacture that comprises acombination of specific purpose and general purpose hardware/software,or the like. In each case, the program code and hardware can be createdusing standard programming and engineering techniques, respectively.

Moreover, the processing unit 338 may comprise a single processing unit,or be distributed across one or more processing units in one or morelocations, e.g., on a client and server. Similarly, the memory 316and/or the storage system 332 can comprise any combination of varioustypes of data storage and/or transmission media that reside at one ormore physical locations. Further, I/O interfaces 324 can comprise anysystem for exchanging information with one or more of the externaldevice 328. Still further, it is understood that one or more additionalcomponents (e.g., system software, math co-processing unit, etc.), notshown, can be included in the computer 322.

One embodiment performs process steps of the invention on asubscription, advertising, and/or fee basis. That is, a service providercould offer to create, maintain, and support, etc., a computerinfrastructure, such as the network computer infrastructure 326 thatperforms the process steps of the invention, for one or more customers.In return, the service provider can receive payment from the customer(s)under a subscription and/or fee agreement and/or the service providercan receive payment from the sale of advertising content to one or morethird parties.

In still another embodiment, the invention provides acomputer-implemented method for executing one or more of the processes,systems and articles according to the present invention as describedabove. In this case, a computer infrastructure, such as the computerinfrastructure 326, can be provided and one or more systems forperforming the process steps of the invention can be obtained (e.g.,created, purchased, used, modified, etc.) and deployed to the computerinfrastructure. To this extent, the deployment of a system can compriseone or more of: (1) installing program code on a computing device, suchas the computers/devices 322/336, from a computer-readable medium; (2)adding one or more computing devices to the computer infrastructure; and(3) incorporating and/or modifying one or more existing systems of thecomputer infrastructure to enable the computer infrastructure to performthe process steps of the invention.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof. As used herein, it is understood thatthe terms “program code” and “computer program code” are synonymous andmean any expression, in any language, code or notation, of a set ofinstructions intended to cause a computing device having an informationprocessing capability to perform a particular function either directlyor after either or both of the following: (a) conversion to anotherlanguage, code or notation; and/or (b) reproduction in a differentmaterial form. To this extent, program code can be embodied as one ormore of: an application/software program, component software/a libraryof functions, an operating system, a basic I/O system/driver for aparticular computing and/or I/O device, and the like.

Certain examples and elements described in the present specification,including in the claims and as illustrated in the Figures, may bedistinguished or otherwise identified from others by unique adjectives(e.g. a “first” element distinguished from another “second” or “third”of a plurality of elements, a “primary” distinguished from a “secondary”one or “another” item, etc.) Such identifying adjectives are generallyused to reduce confusion or uncertainty, and are not to be construed tolimit the claims to any specific illustrated element or embodiment, orto imply any precedence, ordering or ranking of any claim elements,limitations or process steps.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

1. A method for collection of data or information from a dynamic andadaptive target group selection, the method comprising: selecting a setof experts that each have higher value metrics scores relative to valuemetric scores of other unselected experts of a population of knownexperts and that are indicative of an expertise relevant to at least onequestion in a plurality of survey questions; sending the plurality ofsurvey questions to the selected set of experts; expanding the selectedset of experts by adding experts from the population of known expertsthat are noted in referrals in answers received in response to the sentquestions; automatically updating the value metric scores of theexpanded set of experts via a programmable device as a function of atimeliness of a response received from an associated expert and a degreeof correlation of an answer within said response received from theassociated expert to an expected answer; updating the plurality ofsurvey questions by removing questions satisfied by response answersfrom the selected set of experts; and repeating steps of selecting anupdated set of experts that each have higher value metrics scoresrelative to value metric scores of other unselected experts of theselected set and that are indicative of an expertise relevant to atleast one question in the updated plurality of survey questions, sendingthe updated plurality of survey questions to the updated set of experts,expanding the updated set of experts by adding experts noted inreferrals in answers received to the updated plurality of questions,automatically updating the value metric scores of the expanded updatedset of experts via the programmable device as the function of thetimeliness of the responses received and the degree of correlation ofthe answers to the expected answers, and removing satisfied questionsfrom the updated plurality of survey questions, until each question inthe updated plurality of survey questions is satisfied and removed. 2.The method of claim 1, wherein the automatically the updating valuemetric scores of the expanded set of experts via the programmable deviceas the function of the timeliness of the response received from theassociated expert comprises: increasing the value metric score of afirst expert as a function of an elapsed time of a response of the firstexpert to a sent survey question that is less than a normal responsetime; and decreasing the value metric score of a second expert inresponse to a failure to receive a response to a sent survey questionfrom the second expert within an allowable time for response to the sentsurvey question.
 3. The method of claim 2, further comprising: sending areminder to the second expert in response to the failure to receive theresponse to the sent survey question from the second expert within thethreshold allowable time for response; and removing the second expertfrom the updated set of experts in response to a failure to receive aresponse to the reminder within an allowable time for response to thereminder.
 4. The method of claim 2, wherein the automatically updatingvalue metric scores of the expanded set of experts via the programmabledevice is further a function of a number of referrals to an expert fromother experts, further comprising: increasing the value metric score ofa third expert if multiple experts refer to the third expert as anappropriate person to answer a given question of the survey; anddecreasing the value metric score of a fourth expert relative to a fifthexpert if the fourth expert has relatively fewer references to the givenquestion of the survey relative to the fifth expert.
 5. The method ofclaim 1, wherein the expected answer is a hypothesis answer or a meananswer of a totality of answers received from the experts with respectto a sent question, the method further comprising: determining that ananswer received from a sixth expert has a degree of correlation to theexpected answer that is outside of an acceptable variance; and labelingthe sixth expert an outlier expert; or decreasing a value metric scoreof the sixth expert.
 6. The method of claim 5, wherein the updating theplurality of survey questions by removing questions satisfied byresponse answers from the selected set of experts comprises: determiningthat a question has been satisfied with a satisfactory answer uponreceiving a total number of responses that meets a threshold number ofresponses; or determining that a question has been satisfied with asatisfactory answer if an answer to a question is within a thresholdvariance of correlation to the expected answer.
 7. The method of claim6, further comprising: determining that a question has been satisfiedwith a satisfactory answer upon receiving a response from a non-outlierexpert and a response from an outlier expert.
 8. A system, comprising: aprocessing unit, computer readable memory and a computer readablestorage system; first program instructions to select a set of expertsthat each have higher value metrics scores relative to value metricscores of other unselected experts of a population of known experts andthat are indicative of an expertise relevant to at least one question ina plurality of survey questions and to expand the selected set ofexperts by adding experts from the population of known experts that arenoted in referrals in answers received in response to questions of aplurality of survey questions sent to the selected set of experts;second program instructions to determine a degree of correlation of ananswer within said response received from the associated expert to anexpected answer; third program instructions to update the value metricscores of the experts as a function of a timeliness of the responsereceived from an associated expert and of the determined degree ofcorrelation of the answer; and fourth program instructions to update theplurality of survey questions by removing questions satisfied byresponse answers from the selected set of experts and send out theupdated plurality questions to the selected expert sets until eachquestion in the updated plurality of survey questions is satisfied andremoved as determined by iterative repetitive application of the first,second and third instructions; and wherein the first, second, third andfourth program instructions are stored on the computer readable storagesystem for execution by the processing unit via the computer readablememory.
 9. The system of claim 8, wherein the third program instructionsto update the value metric scores of experts are further to: increasethe value metric score of a first expert as a function of an elapsedtime of a response of the first expert to a sent survey question that isless than a normal response time; and decrease the value metric score ofa second expert in response to a failure to receive a response to a sentsurvey question from the second expert within an allowable time forresponse to the sent survey question.
 10. The system of claim 9, whereinthe third program instructions to update the value metric scores ofexperts are further to: send a reminder to the second expert in responseto the failure to receive the response to the sent survey question fromthe second expert within the threshold allowable time for response; andremove the second expert from the updated set of experts in response toa failure to receive a response to the reminder within an allowable timefor response to the reminder.
 11. The system of claim 9, wherein thethird program instructions to update the value metric scores of expertsas a function of a number of referrals to an expert from other expertsare further to: increase the value metric score of a third expert ifmultiple experts refer to the third expert as an appropriate person toanswer a given question of the survey; and decrease the value metricscore of a fourth expert relative to a fifth expert if the fourth experthas relatively fewer references to the given question of the surveyrelative to the fifth expert.
 12. The system of claim 8, wherein theexpected answer is a hypothesis answer or a mean answer of a totality ofanswers received from the experts with respect to a sent question, andwherein the second and third program instructions are further to:determine that an answer received from a sixth expert has a degree ofcorrelation to the expected answer that is outside of an acceptablevariance; and label the sixth expert an outlier expert; or decrease avalue metric score of the sixth expert.
 13. The system of claim 12,wherein the second program instructions are further to: determine that aquestion has been satisfied with a satisfactory answer upon receiving atotal number of responses that meets a threshold number of responses; ordetermine that a question has been satisfied with a satisfactory answerif an answer to a question is within a threshold variance of correlationto the expected answer.
 14. The system of claim 13, wherein the secondprogram instructions are further to: determine that a question has beensatisfied with a satisfactory answer upon receiving a response from anon-outlier expert and a response from an outlier expert.
 15. A computerprogram product for collection of data or information from a dynamic andadaptive target group selection, the computer program productcomprising: a computer readable storage medium; first programinstructions to select a set of experts that each have higher valuemetrics scores relative to value metric scores of other unselectedexperts of a population of known experts and that are indicative of anexpertise relevant to at least one question in a plurality of surveyquestions and to expand the selected set of experts by adding expertsfrom the population of known experts that are noted in referrals inanswers received in response to questions of a plurality of surveyquestions sent to the selected set of experts; second programinstructions to determine a degree of correlation of an answer withinsaid response received from the associated expert to an expected answer;third program instructions to update the value metric scores of theexperts as a function of a timeliness of the response received from anassociated expert and of the determined degree of correlation of theanswer; and fourth program instructions to update the plurality ofsurvey questions by removing questions satisfied by response answersfrom the selected set of experts and send out the updated plurality ofquestions to the selected expert sets until each question in the updatedplurality of survey questions is satisfied and removed as determined byiterative repetitive application of the first, second and thirdinstructions; and wherein the first, second, third and fourth programinstructions are stored on the computer readable storage medium.
 16. Thecomputer program product of claim 15, wherein the third programinstructions to update the value metric scores of experts are furtherto: increase the value metric score of a first expert as a function ofan elapsed time of a response of the first expert to a sent surveyquestion that is less than a normal response time; and decrease thevalue metric score of a second expert in response to a failure toreceive a response to a sent survey question from the second expertwithin an allowable time for response to the sent survey question. 17.The computer program product of claim 16, wherein the third programinstructions to update the value metric scores of experts are furtherto: send a reminder to the second expert in response to the failure toreceive the response to the sent survey question from the second expertwithin the threshold allowable time for response; and remove the secondexpert from the updated set of experts in response to a failure toreceive a response to the reminder within an allowable time for responseto the reminder.
 18. The computer program product of claim 16, whereinthe third program instructions to update the value metric scores ofexperts as a function of a number of referrals to an expert from otherexperts are further to: increase the value metric score of a thirdexpert if multiple experts refer to the third expert as an appropriateperson to answer a given question of the survey; and decrease the valuemetric score of a fourth expert relative to a fifth expert if the fourthexpert has relatively fewer references to the given question of thesurvey relative to the fifth expert.
 19. The computer program product ofclaim 15, wherein the expected answer is a hypothesis answer or a meananswer of a totality of answers received from the experts with respectto a sent question, and wherein the second and third programinstructions are further to: determine that an answer received from asixth expert has a degree of correlation to the expected answer that isoutside of an acceptable variance; and label the sixth expert an outlierexpert; or decrease a value metric score of the sixth expert.
 20. Thecomputer program product of claim 19, wherein the second programinstructions are further to: determine that a question has beensatisfied with a satisfactory answer upon receiving a total number ofresponses that meets a threshold number of responses; or determine thata question has been satisfied with a satisfactory answer if an answer toa question is within a threshold variance of correlation to the expectedanswer.
 21. The computer program product of claim 20, wherein the secondprogram instructions are further to: determine that a question has beensatisfied with a satisfactory answer upon receiving a response from anon-outlier expert and a response from an outlier expert.